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Abstract 



This study investigated school-level benefits of using the PLAN program over time. 
Outcome variables included the average gain in ACT Assessment scores, the proportion of 
students whose ACT scores achieved college readiness benchmarks, and the proportion of 
students who took particular patterns of college preparatory coursework in high school. Schools 
were the unit of analysis and were divided into four categories of PLAN participation. Over time, 
schools that consistently tested all of their sophomores with PLAN had increases in average ACT 
Composite scores .3 1 units higher than the average increases of schools that did not use PLAN. 
Schools that used PLAN also typically had greater positive increases than PLAN non-users with 
respect to the other outcome variables. Moreover, gains in the outcome variables increased with 
the number of years schools tested all of their sophomores. 
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School-level Benefits of Using PLAN Over Time 



ACT developed the Educational Planning and Assessment System® (EPAS) to help 
students prepare for high school and the transition to college and work after high school. EPAS 8 
consists of three different assessments that are administered to students at certain points in 
middle school and high school. EXPLORE is usually taken in grade 8 or 9, PLAN in grade 10, 
and the ACT Assessment in grade 11 or 12. All three assessments are designed to measure 
students’ academic knowledge and skills at each of the three education points. The EPAS 
program provides a longitudinal overview for educational and career planning, instructional 
support, assessment, and evaluation. 

Previous research supports the use of EPAS to assess student achievement (e.g., Roberts 
& Noble, 2004; Schiel, Pommerich, & Noble, 1996). These studies showed the contributions of 
courses taken between grades 8 and 10, and grades 10, 1 1 and 12, on PLAN and/or ACT scores, 
while statistically controlling for prior achievement. In addition, research has shown that students 
from schools participating in EPAS achieve higher PLAN and ACT scores and are more likely to 
take rigorous coursework when compared to students from schools not participating in EPAS 
(Noble, 2003). However, this prior research focused only on one period in time and did not 
consider at improvement over time or the length of time schools had used either EXPLORE or 
PLAN. 

The purpose of this study is to demonstrate the benefits to schools of the PLAN program 
when used over time. Only the PLAN/ACT portion of EPAS was investigated here to maximize 
the time span to be studied and the number of user and non-user schools. Benefits associated 
with PLAN use over time included increases in ACT test scores, as well as increases in college 
preparatory coursework taken and readiness for college-level work. 
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Data 

The data consisted of ACT Assessment records of students tested between 1994 and 
2003. Their high schools were categorized according to their use of PLAN. The PLAN/ACT 
matched history for each year was matched to the relevant ACT history to identify PLAN user 
and non-user schools. For the matched PLAN/ACT files samples sizes ranged from 374,526 
students (9,234 schools) in 1993-94 to 438,076 students (6,307 schools) in 2002-03. Sample 
sizes for the ACT history files ranged from 891,992 students (19,886 schools) in 1993-94 to 
1,175,059 students (23,268 schools) in 2002-03. To be included in the study, PLAN user schools 
could have started using PLAN at any time, but were required to have used PLAN from 2000 
through 2003. Moreover, all PLAN user schools were required to have consistently used PLAN 
over time (i.e., used every year), once they implemented PLAN. Non-user schools were required 
to have never used PLAN at any time between 1994 and 2003. 

PLAN user schools were then classified by type of use. A distinction was made between 
schools that tested all their sophomores with PLAN and schools that only tested volunteer 
students. (This is an important distinction because volunteer students tend to be higher-achieving 
students.) Four PLAN use categories of schools were identified: 

1. Schools that consistently tested all of their sophomores with PLAN, once they started 
using PLAN (consistent census users; n = 1,145). 

2. Schools that vacillated over time between testing all of their sophomores and testing a 
volunteer group of sophomores (inconsistent census users; n = 2,237). 

3. Schools that consistently did not census-test (i.e., tested student volunteers)(non-census 
users; n = 2,087). 

4. Schools that never used PLAN (non-users; n = 1,565). 
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Schools with an average of less than 10 students taking the ACT Assessment from 2000-03 were 
not included in the data set. 

Sampling weights were calculated for each school to account for differences in the 
statistical precision associated with schools with different sample sizes. Unweighted mean school 
sample sizes by PLAN use category were as follows: 



Type 


Mean sample 
size 


SD 


Consistent census users 


96.22 


106.32 


Inconsistent census users 


82.16 


93.09 


Non-census users 


102.40 


92.43 


Non-users 


26.11 


20.88 



Weights were calculated using the number of ACT Assessment tested students in the school, as 
follows: 

wj = (: nj/ni.)*N , 

where w y is the weight for school j, 

riij is the number of students i within school j, 

rij. is the number of students i across all schools, and 

N is the number of schools. 

School-level variables were created by calculating means and proportions for the student- 
level variables from the ACT record. Means and proportions were also calculated within gender 
and racial/ethnic groups. These variables included: 

• gender, 

• ethnicity, 

• grade level, 

• family income (on a zero to nine scale, with nine being the highest), 





4 



• ACT scores, 

• educational plans, 

• core participation (three years of mathematics, social studies, and science and four 
years of English), 

• college preparatory curriculum, 

• major and occupational plans, 

• responses to needs items, and 

• high school coursework taken or planned. 

The ACT school level data were then matched with the Market Data Retrieval data files 
(MDR; Shelton, Connecticut) to obtain other school characteristics. These variables included 
accreditation region of the country, type of school (public/private), school location 
(urban/suburban/rural), and per-grade enrollment. Categorical school characteristics were either 
dummy-coded or effect-coded for analysis. Accreditation regions were recoded to combine 
certain regions: the four resulting regions were Southern, Northwest and West, North Central, 
and Middle States and New England. 

Method 

Unweighted means, standard deviations, and sample sizes were calculated by type of 
PLAN use for average ACT Composite and all school characteristics. For each outcome and 
school, three-year averages were used instead of single years to stabilize the results. Then, 
average gains were calculated by taking the difference between the 2001-03 averages and the 
averages for three years prior to PLAN implementation (baseline years). For the non-user group, 
the baseline years were 1997-99. 
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ACT college readiness benchmarks were defined as meeting or exceeding an ACT 
English score of 18, an ACT Mathematics score of 22, and an ACT Science score of 24 (Allen & 
Sconing, in press). These benchmarks reflect a student’s likely success in college English 
Composition, college Algebra, and college Biology. 

Outcome Variables 

Average gains for each outcome were regressed on PLAN participation category, number 
of years of census testing, and school characteristic variables. All analyses were conducted for 
the total group, for racial/ethnic groups (African American, Caucasian American, Hispanic, 
Asian American, and Other), and for gender groups. Outcome variables were evaluated for each 
of these groups, and included the following: 

1. ACT scores 

2. Preparation for college-level work (meeting or exceeding ACT benchmark scores 
in English, Mathematics, and Science) 

3. Sureness of career and major choices 

4. College-bound vs. non-college-bound 

5. Need for help in writing, mathematics, reading, study skills, educational plans, or 
career plans 

6. Taking/not taking the core curriculum 

7. Taking (or taking/planning to take) specific mathematics or science course 
patterns (4 and 3, respectively) 

8. College preparatory curriculum vs. other 
School characteristic variables included the following: 

1 . Location (rural, urban, suburban) 
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2. Per grade enrollment 

3. Average family income in the school 

4. Region 

5. Length of participation in EPAS 

6. Type of school (public vs. private) 

7. Proportion of students taking the ACT Assessment 
Regression Analyses 

Initial regression models were developed using school-level average ACT gain scores 
and predictor variables used in prior research (ACT, Inc., in press; Noble, 2003), including 
school characteristics (e.g., location, region), proportion of ACT-tested students, proportion of 
juniors, and average family income. Variable and model selection was based on the statistical 
significance (p<.001) of the regression coefficients, model statistical significance (p<.001), R" 
value, and standard error of estimate (SEE). Most of the regression coefficients were statistically 
significant due to the large sample size. 

Final model development. Model fit was determined using the model for average ACT 
Composite score gains. Once a final model was established for this outcome, the same model 
was developed for all other outcome variables for the total group, and by gender and racial/ethnic 
group. Predictor variables included in the final model were: 

• dummy-coded variables identifying type of school participation, 

• the number of years a school census-tested with PLAN, 

• an indicator variable identifying schools in one specific state in the North Central 
accrediting region, 

• 



school type (public/private), 
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• average family income within a school, 

• proportion of ACT tested students within a school, 

• school location (urban/suburban/rural), and 

• recoded accreditation region of the school. 

The variable representing schools from one state was used because the percentages of schools by 
PLAN use category differed substantially from those of other states. Inclusion of all of these 
variables would statistically control for school characteristics and eliminate their confounding 
with PLAN use. 

Results 

Unweighted descriptive statistics for schools using PLAN (consistent census, inconsistent 
census, non-census) and not using PLAN are provided in Table 1. The table includes only the 
variables used in the final regression model. 

Average ACT Composite scores for the most recent three years of PLAN use were 
similar to or slightly higher than the average for three years prior to use for all three types of 
PLAN user schools. The 2002-03 national average ACT Composite score was 20.8, and only 
PLAN consistent census user schools had average Composite scores higher than the national 
average. PLAN consistent census schools had slightly higher average family incomes and higher 
proportions of ACT-tested students than non-census and non-user schools. Consistent census, 
inconsistent census, and non-census schools were predominantly from the North Central 
accrediting region. 




TABLE 1 



Descriptive Statistics for Schools by PLAN Participation 





Means (SD) and proportions | 




Consistent 


Inconsistent 






School level statistics 


census 


census 


Non-census 


Non-user , 


Avg. ACT Composite 
recent 3 years 


21.2(1.84) 


20.7(1.98) 


20.7 (1.77) 


20.1(2.51) 


Avg. ACT Composite 
prior 3 years 


20.8 (1.58) 


20.4(1.67) 


20.6 (1.50) 


20.7 (2.42) 


Avg. years PLAN use 


9.49 


9.55 


9.59 


0 ' 


Avg. years census use 


6.05 


4.12 


0 


o ! 


Avg. family income 


5.00 


4.68 


4.72 


4.84 ! 


Pet. of high school 
graduates ACT tested 


55 


54 


40 


18 


Pet. public 


69 


74 


97 


86 ' 


Pet. specific state 


12 


10 


9 


1 


Pet. rural 


53 


57 


57 


41 


suburban 


27 


23 


29 


35 


urban 


20 


20 


14 


24 1 


Pet. Middle/New England 


3 


3 


4 


37 


North Central 


73 


62 


61 


15 


Northwest/West 


3 


5 


10 


21 


Southern 


21 


30 


25 


26 



Regression Results 

Schools that used PLAN typically had greater positive increases over time than PLAN 
non-users for all outcome variables except the following: 

• meeting or exceeding the ACT benchmark score in English; 

• college-bound vs. non-college-bound; 

• need for help in mathematics, reading, study skills, and education or career plans; 

• college preparatory curriculum vs. other; and 

• taking (or planning to take) certain mathematics and science course patterns. 

Moreover, PLAN user schools that consistently census-tested and PLAN schools that 
consistently tested volunteer students had greater increases than PLAN schools that 
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inconsistently census tested their students. Outcome variable gains also increased with the 
number of years of census testing. 

Regression coefficients for each independent variable in a regression model reflect the 
average increase in the outcome variable associated with a one-unit increase in an independent 
variable, given all other variables in the model. Outcomes were interpreted in terms of the 
increase in each outcome variable associated with PLAN use. In all cases the comparison group 
was the non-user group. 

All regression models were statistically significant (p < .001), except for sureness of 
major and occupation and mathematics coursework patterns taken and planned for Asian and 
Hispanic students. Model R values ranged from .20 to .25 for average score gains, from .15 to 
.24 for percentages of students meeting the college readiness benchmarks, and less than .10 for 
percentages of students needing help in particular areas and percentages of students taking 
specific course patterns. All differences by PLAN use category described below were 
statistically significant (p < .05), unless otherwise identified. 

ACT score results. The average ACT Composite score increase for schools that 
consistently census-tested with PLAN over time was .31 score units higher than that for schools 
not using PLAN. Average score increases for inconsistent census users and non-census users 
were somewhat lower (.24 and .27, respectively). Greater gains over time in ACT Composite 
scores were found for females than for males from schools using PLAN, regardless of how 
PLAN was implemented, compared to males and females from non-user schools (see Figure 1). 
This pattern also held true for ACT Reading and Science scores. 
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FIGURE 1. Average Increase in ACT Composite Score, by PLAN 
Use Category and Gender 
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■ Consistent Census □ Inconsistent Census □ Non-Census 



For Other Race students, PLAN use was associated with a .77 to .99 ACT Composite 
score increase over time, regardless of how the school implemented PLAN, compared to PLAN 
non-user schools. This pattern was found for all ACT subject tests. Upon further investigation, it 
was found that the majority of PLAN and ACT-tested students that responded as Other Race on 
the ACT Assessment had identified themselves as Caucasian or Asian American students when 
they took PLAN. 

On average, PLAN use was associated with an ACT Mathematics score increase of .58 to 
.63 score units compared to PLAN non-user schools, with consistent census user schools having 
the greatest average increase. ACT Mathematics scores for males at PLAN user schools were .69 
to .76 score units higher, and scores for females were .53 to .61 score units higher than those at 
PLAN non-user schools (see Figure 2). Score increases were somewhat higher for males than for 
females for all three PLAN user groups. 
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FIGURE 2. Average Increase in ACT Mathematics Score, by 
PLAN Use Category and Gender 
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■ Consistent Census EH Inconsistent Census □ Non-Census 



Hispanic students from consistent census user schools had higher PLAN Mathematics 
score increases than those at non-user schools (by .58 score units). High gains were also found 
for Caucasian American students (.87) and Other Race students (.99) from consistent census user 
schools. 

College readiness benchmark results. PLAN user schools had slightly higher increases in 
average percentages of students prepared for college-level work, compared to PLAN non-users, 
with a 5% increase for students meeting the ACT Mathematics benchmark of 22 and a 3% 
increase for students meeting the ACT Science benchmark of 24. These findings were consistent 
across gender groups. 

Average increases over time for racial/ethnic groups in percentage of students meeting 
the ACT Mathematics benchmarks are shown in Figure 3. On average, schools consistently using 
PLAN census testing had higher increases in the percentage of Hispanic students (4%) and 
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African American students (3%) meeting the Mathematics readiness benchmark, compared to 
PLAN non-user schools. In addition, PLAN user schools had an average increase of 6% to 7% of 
Other Race students meeting the ACT Mathematics and Science college readiness benchmarks, 
compared to PLAN non-user schools. 

FIGURE 3. Average Increase in Percentage of Students Meeting the ACT Mathematics 
Benchmark, by PLAN Use Category and Race/Ethnicity 




■ Consistent Census □ Inconsistent Census □ Non-Census 



Need for help results. PLAN user schools had a greater decrease in the average 
percentage of students needing help with Writing than did PLAN non-user schools (4% to 5% 
decrease). This decrease was larger for females (6% to 7%) than for males (3% to 4%), and was 
especially true for all racial/ethnic groups except Caucasian American students (see Figures 4 



and 5). 
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FIGURE 4. Decrease in Average Percentage of Students Needing Help with Writing, by PLAN 

Use Category and Gender 
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FIGURE 5. Decrease in Average Percentage of Students Needing Help with Writing, by PLAN 

Use Category and Race/Ethnicity 
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Core coursework results. PLAN user schools had a higher increase in average percentage 
of students taking core coursework than PLAN non-user schools (9% to 11%). This increase was 
higher for females (10% to 12%) than for males (8%), as shown in Figure 6. The average 
increase in the percentage of African American, Caucasian American, Hispanic, and Other Race 
students taking core coursework was also higher for PLAN user schools, as shown in Figure 7. 



FIGURE 6. Increase in Average Percentage of Students Taking Core Coursework, by PLAN Use 

Category and Gender 




■ Consistent Census E3 Inconsistent Census □ Non-Census 
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FIGURE 7. Increase in Average Percentage of Students Taking Core Coursework, by PLAN Use 

Category and Race/Ethnicity 
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■ Consistent Census 0 Inconsistent Census □ Non-Census 



Specific course pattern results. The average increase in the percentage of students who 
had taken or were currently taking General Science, Biology, and Chemistry was 5% higher for 
PLAN user schools than for PLAN non-user schools, irrespective of type of PLAN use. This 
increase was the same for males and females. There was an especially pronounced increase for 
Hispanic and African American students (see Figure 8). This same trend (2% higher for PLAN 
user schools) was also observed for students who had taken or planned to take this course 
pattern. Students at PLAN user schools who had taken or planned to take this course pattern plus 
a Physics course also had higher increases (6% to 7%) consistent with the above trend, 
irrespective of type of use. This increase was especially high for Hispanic students (5% to 9%). 
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FIGURE 8. Increase in Average Percentage of Students Taking a Specific Science Course, by 

PLAN Use Category and Race/Ethnicity 
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■ Consistent Census SI Inconsistent Census □ Non-Census 



In addition, average increases in the percentage of students who had taken or were 
currently taking specific mathematics course patterns were 1% to 3% higher for PLAN user 
schools, regardless of type of use, than for PLAN non-user schools. Similar increases were seen 
for males and females across all mathematics course patterns. For course pattern 1 (Algebra 1, 
Algebra 2, Geometry, Trigonometry, and Calculus), increases were highest for Hispanic and 
Asian students (1% to 3%). Increases for Hispanic students were also high for course pattern 2 
(Algebra 1, Algebra 2, Geometry, and Trigonometry). For course pattern 3 (Algebra 1, Algebra 
2, Geometry, Trigonometry, and Other Advanced Mathematics), increases across racial/ethnic 
groups were fairly consistent. The same trend was observed for students who had taken or 
planned to take this course pattern. 

Discussion and Implications 

Results of this study indicate that schools benefit from using PLAN in terms of score 
gains over time and increases in the percentage of students meeting or exceeding college 
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readiness benchmarks in mathematics and science. In addition, schools using PLAN had 
increased percentages of students taking core coursework and challenging course patterns, and 
decreased percentages of students needing help with writing, compared to non-user schools. 
Moreover, longer time spans of census testing were associated with greater gains in outcome 
variables. Schools that were consistent census users or non-census users outperformed 
inconsistent census users for all outcomes except the percentage of students taking the core 
curriculum. These findings indicate that it is beneficial for schools to use the PLAN/ACT portion 
of EPAS in a consistent manner over time. Consistency could be a good indicator of how firmly 
entrenched the program is in the school and therefore, how much the program affects outcomes 
in a desirable manner. 

Although the results indicate that consistency over time is a desirable implementation 
goal for EPAS, this study did not look specifically at how schools were using PLAN, only at the 
type and length of use. Data were not available on how schools used PLAN for student planning, 
instructional support, assessment, and evaluation. The benefits associated with the 
implementation of the EPAS program are directly related to how it is used. Certain uses of 
PLAN information could, for example, result in greater achievement. 

Differences among racial/ethnic groups depended on the outcome variable of interest. In 
general, greater gains were found for Hispanic, Caucasian American, and Other Race students 
than for Asian American or African American students. This may be due, in part, to the 
differences in sample sizes by racial/ethnic group; sample sizes for African American and Asian 
American students were the smallest of all of the racial/ethnic groups. 

It may also be beneficial to target certain subject areas to determine whether level of 
implementation differs across subjects. Although increases were observed in the percentage of 
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students meeting or exceeding the mathematics and science benchmarks, there was no increase 
observed for the English benchmarks. English could be targeted to investigate why reaching the 
English benchmark was not differentially affected by PLAN use. 

With today’s emphasis on educational planning and accountability, the EPAS program 
offers educators a viable option for tracking student performance and advancement. Use of the 
program is also related to reducing achievement gaps across racial/ethnic and gender groups. 
Schools that use PLAN consistently over time show improvements that non-user schools do not. 
This argues for schools to start using PLAN in a consistent manner (whether census testing or 
not) over consecutive years as a way to improve student performance. 
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